@richenc This was a long-standing gap in our Airflow plugin, so thanks for taking a stab at it!

We'll be reviewing it soon.

@hsheth2 I noticed the v2 plugin released recently. I was wondering how that affects my PR.

@richenc this PR still applies for the v1 plugin and we can merge review/merge it in around v1 if we want. However, the v2 plugin uses the Airflow listener API (instead of the somewhat hacky policy mechanism that v1 uses), so we should be able to detect UP_FOR_RETRY events more reliably with v2.

The external sensor changes seem reasonable and are part of common code between v1 and v2, so that logic should be ok to merge regardless

@richenc this PR still applies for the v1 plugin and we can merge review/merge it in around v1 if we want. However, the v2 plugin uses the Airflow listener API (instead of the somewhat hacky policy mechanism that v1 uses), so we should be able to detect UP_FOR_RETRY events more reliably with v2.

The external sensor changes seem reasonable and are part of common code between v1 and v2, so that logic should be ok to merge regardless

@hsheth2 good to hear that the listener API is being used! I think that means it might be possible to emit the data flow runs as dataprocessinstances! I'm going to look into the v2 plugin so I can migrate and contribute to it in the future.

I just rebased my branch to fix the conflicts. Please let me know if there is anything else I need to do before this can be merged.

The code looks good

Do you have a sense for how we'd implement something similar with the v2 plugin (using the listener API)?

thanks @hsheth2! At the moment, I'm not sure but I was thinking about taking a crack at it once I had some time to test the v2 plugin in our local environment in preparation for migrating.